SharpSpider: Spidering the Web through Web Services
نویسندگان
چکیده
Web search engines have become an indispensable utility for Internet users. In the near future, however, Web search engines will not only be expected to provide quality search results, but also to enable applications to search and exploit their index repositories directly. We present here SharpSpider, a distributed, C# spider designed to address the issues of scalability, decentralisation and continuity of a Web crawl. Fundamental to the design of SharpSpider is the publication of an API for use by other services on the network. Such an API grants access to a constantly refreshed index built after successive crawls of the Web.
منابع مشابه
A procedure for Web Service Selection Using WS-Policy Semantic Matching
In general, Policy-based approaches play an important role in the management of web services, for instance, in the choice of semantic web service and quality of services (QoS) in particular. The present research work illustrates a procedure for the web service selection among functionality similar web services based on WS-Policy semantic matching. In this study, the procedure of WS-Policy publi...
متن کاملImage flip CAPTCHA
The massive and automated access to Web resources through robots has made it essential for Web service providers to make some conclusion about whether the "user" is a human or a robot. A Human Interaction Proof (HIP) like Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) offers a way to make such a distinction. CAPTCHA is a reverse Turing test used by Web serv...
متن کاملDeep Web Mining through Web Services
With the increase in Web-based databases and dynamically-generated Web pages, the concept of the “deep Web” has arisen. The deep Web refers to Web content that, while it may be freely and publicly accessible, is stored, queried, and retrieved through a database and one or more search interfaces, rendering the Web content largely hidden from conventional search and spidering techniques. These me...
متن کاملQoS-Based web service composition based on genetic algorithm
Quality of service (QoS) is an important issue in the design and management of web service composition. QoS in web services consists of various non-functional factors, such as execution cost, execution time, availability, successful execution rate, and security. In recent years, the number of available web services has proliferated, and then offered the same services increasingly. The same web ...
متن کاملQoS-based Web Service Recommendation using Popular-dependent Collaborative Filtering
Since, most of the organizations present their services electronically, the number of functionally-equivalent web services is increasing as well as the number of users that employ those web services. Consequently, plenty of information is generated by the users and the web services that lead to the users be in trouble in finding their appropriate web services. Therefore, it is required to provi...
متن کامل